The approaches described in this section could be pursued, but are not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated herein, the approaches described in this section are not prior art to the claims in this application and are not admitted to be prior art by inclusion in this section.
Common user interaction with multimedia devices such as computers, smartphones and tablets usually rely on user inputs like keyboard events, mouse events and screen touch events such as pinch, zoom, touch, etc.
Other interactions are also becoming common like vocal and gestural commands, which are generally used as indirect triggers, as an analysis of a related context is usually required to engage the action expected by the user.
This context may be commonly related and possibly limited to the specific application currently used by the user. In this case, the considered context includes for example the recent queries made by the user or the last opened menu.
The context may also include the immediate user environment, such as the physical location of the user in a room and the items present in the room, as the user can perceive them through his naked eyes or using a see-through device or a head-mounted display (HMD). An example of a context including real objects in the immediate user environment is shown in FIG. 1.
In the example of FIG. 1, a user 2 equipped with a head-mounted display 4, can interact with a multitude of real objects located in a room 6. For instance, these objects include objects placed on a table 7, such as a camera 8 and a newspaper 10.
Moreover, in some cases such as altered reality or virtual reality applications, the context may include real and/or virtual object(s) from the surroundings of the user. An example of such context is shown in FIG. 2.
By altered reality, it is meant both the augmented reality (also called mixed reality) case i.e. real objects plus some virtual objects present in a scene and the case where real objects are altered, for example in size, color, presence, or location, as seen by the user either directly or by using a device such as see-through glasses.
In the example of FIG. 2, the user 2 equipped with the head-mounted display 4, can interact with the real objects 8, 10. The user 2 can also interact with virtual objects located in the room 6. For instance, the virtual objects include a globe 12, a gamepad 14, an equalizer 16, a character 18 and a gear 20. These virtual objects may be related to an application in the head-mounted display 4. For instance, the virtual objects may serve as an avatar of available applications, or avatars of elements that may be activated to trigger some related application, for example the gamepad 14 for a game launch.
In the case of multiple real and/or virtual objects in the immediate environment of the user, the user may need to specifically indicate the object that he intends to interact with. This selection of the object needs to be very accurate to avoid a misunderstanding of the user's intent by selecting a wrong object.
In common gaming approaches, such triggering of interaction is limited to a few objects at one time. The number of objects is usually linked to the number of buttons of the gamepad used by the player. Some games, for instance Heavy Rain Move Edition in Play Station 3 and Play Station 4 consoles, use a specific control device and a finely tuned layout of the objects in the play scene.
The document US2013/0328925 describes a system for interpreting user focus on virtual objects in a mixed reality environment. This system includes a see-through, head mounted display device coupled to one or more processing units. The processing units in cooperation with the head mounted display unit(s) can display one or more virtual objects, also referred to as holographic objects, to the user. The user may interact with the displayed virtual objects. Using inference, express gestures and heuristic rules, the system determines which of the virtual objects the user is likely focused on and interacting with. At that point, the system may emphasize the selected virtual object over other virtual objects, and interact with the selected virtual object in a variety of ways.
In this document, the selection of the object that the user wants to interact with is made by determining the position of the user's head or eyes or hand with respect to the virtual object or by determining the movement of the user's hand with respect to the virtual object.
One problem with the prior art solutions is that they do not manage accurately object selection in case of grouped objects in a close neighborhood. Also, the existing solutions cannot handle accurately a partial or a full overlap of real and/or virtual objects from the user viewpoint.
FIG. 3 shows an example of a realistic augmented scene with numerous virtual object(s). Such scene would not be easily managed through existing systems.
The objects present in the scene of FIG. 3 are the same as those present in the scene of FIG. 2, with the camera 8 so close to the newspaper 10 that it covers it partially.
How would the user 2 be able to easily trigger some interaction, for instance by moving his/her hand 22, with a specific object in the scene of FIG. 3, such as the camera 8 or the newspaper 10 on the table 7, when they are so close, or even partially or fully occluding for instance?